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Abstract. The conformance testing problem for dynamical systems asks, given two dynamical models 
(e.g., as Simulink diagrams), whether their behaviors are “close” to each other. In the semi-formal 
approach to conformance testing, the two systems are simulated on a large set of tests, and a metric, 
defined on pairs of real-valued, real-timed trajectories, is used to determine a lower bound on the 
distance. We show how the Skorkhod metric on continuous dynamical systems can be used as the 
foundation for conformance testing of complex dynamical models. The Skorokhod metric allows for both 
state value mismatches and timing distortions, and is thus well suited for checking conformance between 
idealized models of dynamical systems and their implementations. We demonstrate the robustness of 
the system conformance quantification by proving a transference theorem: trajectories close under the 
Skorokhod metric satisfy “close” logical properties. Specifically, we show the result for the timed linear 
time logic TLTL augmented with a rich class of temporal and spatial constraint predicates. We provide 
a window-based streaming algorithm to compute the Skorokhod metric, and use it as a basis for a 
conformance testing tool for Simulink. We experimentally demonstrate the effectiveness of our tool in 
finding discrepant behaviors on a set of control system benchmarks, including an industrial challenge 
problem. 


1 Introduction 

A fundamental question in model-based design is conformance testing: whether two models of a 
system are equivalent. For discrete systems, this question is well-studied [2fin7l1SI27j . and there is 
a rich theory of process equivalences based on similarity and bisimilarity. For continuous and hybrid 
systems, however, the state of the art is somewhat unsatisfactory. While there is a straightforward 
generalization of process equivalences to the continuous case, in practice, equivalence notions such 
as bisimilarity are always too strong and most systems are not bisimilar. Since equivalence is a 
Boolean notion, one gets no additional information about the systems other than they are “not 
bisimilar,” and even if two dynamical systems are bisimilar, they may still differ in many properties 
that are of control-theoretic interest. Thus, classical notions for equivalence and conformance have 
been of limited use in industrial practice. 

In recent years, the notion of bisimulation has therefore been generalized to metrics on systems, 
which quantify the distance between them. For example, one approach is that of e-bisimulation, 
which requires that the states of the two systems remain “close” forever (within an e-ball), rather 
than coincide exactly. Under suitable stability assumptions on the dynamics, one can prove re¬ 
sults about e-bisimulation Unfortunately, proving the pre-requisites for the existence of e- 

bisimulations for complex dynamical models, or coming up with suitable and practically tractable 
bisimulation functions, is extremely difficult in practice. In addition, establishing e-bisimulation 
requires full knowledge of the system dynamics making the scheme inapplicable where one system 
is an actual physical component with unknown mathmatical dynamics. Bisimulation notions have 
hence been of limited practical use. 

* This research was funded in part by a Humboldt foundation grant, FCT grant SFRHBPD902672012, and by a 
contract from Toyota Motors. 







Instead, a more pragmatic semi-formal approach has gained prominence in industrial practice. 
In this approach, the two systems are executed on the same input sequences and a metric on finite 
trajectories is used to evaluate the closeness of these trajectories. The key to this methodology is 
the selection of a good metric, with the following properties: 

— Transference. Closeness in the metric must translate to preserving interesting classes of logical 
and functional specifications between systems, and 

— Tractability. The metric should be efficiently computable. 

In addition, there is the more informal requirement of applicability: the metric should classify 
systems, that the engineers consider close, as being close, and conversely. 

A number of metrics have been proposed recently. The simplest is a pointwise metric that 
computes the maximum pointwise difference between two trajectories, sometimes generalized to 
apply a constant time-shift to one trajectory m- Unfortunately, for many practical models, two 
trajectories may be close only under variable time-shifts. This is the case, for example, for two 
dynamical models that may use different numerical integration techniques (e.g., fixed step versus 
adaptive step) or when some component in the implementation has some jitter. Thus, the pointwise 
metric spuriously report large distances for “close” models. More complicated hybrid distances have 
been proposed [T]. The transference properties of these metrics w.r.t. common temporal logics for 
dynamical systems are not yet clear. 

In this work we present a methodology for quantifying conformance between real-valued dy¬ 
namical systems based on the Skorokhod metric HU. The Skorokhod metric allows for mismatches 
in both the trace values and in the timeline, and quantifies temporal and spatial variation of the 
system dynamics under a unifying framework. The distortion of the timeline is specified by a re¬ 
timing function r which is a continuous bijective strictly increasing function from M_|_ to M+. Using 
the retiming function, we obtain the retimed trace x (r(t)) from the original trace x{t). Intuitively, 
in the retimed trace x (r(t)), we see exactly the same values as before, in exactly the same order, 
but the time duration between two values might now be different than the corresponding duration 
in the original trace. The amount of distortion for the retiming r is given by sup^>o|r(f) — t\. Using 
retiming functions, the Skorokhod distance between two traces x and y is defined to be the least 
value over all possible retimings r of: 

max sup |r(t) —t|, sup D(x (r(f)), 7/(f)) , 

\te[o,T] te[o,T] J 

where T is a pointwise metric on values. The Skorokhod distance thus incorporates two components: 
the first component quantifies the timing discrepancy of the timing distortion required to “match” 
two traces, and the second quantifies the value mismatch (in the metric space 0) of the values under 
the timing distortion. The Skorokhod metric was introduced as a theoretical basis for defining the 
semantics of hybrid systems by providing an appropriate hybrid topology m- We now demonstrate 
its usefulness in the context of conformance testing. 

Transference. We show that the Skorokhod metric gives a robust quantification of system con¬ 
formance by relating the metric to TLTL (timed LTL) enriched with (i) predicates of the form 
f{xi,... ,Xn) > 0, as in Signal Temporal Logic, for specifying constraints on trace values; and (ii) 
freeze quantifiers, as in TPTL [ 1 ], for specifying temporal constraints (freeze quantifiers can ex¬ 
press more complex timing constraints than bounded timing constraints, e.g of MTL). This logic 
subsumes the MITL-based logic STL [13]. We prove a transference theorem: flows (and propo¬ 
sitional traces) which are close under the Skorokhod metric satisfy “close” TLTL formulae for a 


rich class of temporal and spatial predicates; where the untimed structure of the formulae remains 
unchanged, only the predicates are enlarged. 

Tractability. We improve on recent polynomial-time algorithms for the Skorokhod metric |23j by 
taking advantage of the fact that, in practice, only retimings that map the times in one trace to 
“close” times in the other are of interest. This enables us to obtain a streaming sliding-window 
based monitoring procedure which takes only 0(W) time per sample, where W is the window size 
(assuming the dimension n of the system to be a constant). 

Usability. Using the Skorokhod distance checking procedure as a subroutine, we have implemented 
a Simulink toolbox for conformance testing. Our tool integrates with Simulink’s model-based design 
flow for control systems, and provides a stochastic search-based approach to find inputs which 
maximize the Skorokhod distance between systems under these inputs. 

We present three case studies from the control domain, including industrial challenge problems; 
our empirical evaluation shows that our tool computes sharp estimates of the conformance distance 
reasonably fast on each of them. Our input models were complex enough that more theoretically 
appealing techniques such as e-bisimulation function generation could not be applied. In particular, 
we demonstrate how two models that only differ in the underlying ODE solver can nevertheless 
deviate enough to invalidate system requirements on settling time. 

We conclude that the Skorokhod metric can be an effective foundation for semi-formal confor¬ 
mance testing for complex dynamical models. Proofs of the theorems are given in the accompanying 
technical report [REF]. 

Related Work. The work of |ll2j is closely related to ours. In it, robustness properties of hybrid 
state sequences are derived with respect to a trace metric which also quantifies temporal and spatial 
variations. Our work differs in the following ways. First, we guarantee robustness properties over 
flows rather than only over (discrete) sequences. Second, the Skorokhod metric is a stronger form 
of the (T, J, (r, e))-closeness degre^^J^for systems which do not have hybrid time); and allows us to 
give stronger robustness transference guarantees. The Skorokhod metric requires order preservation 
of the timeline, which the (T, J, (r, e))-closeness function does not. Preservation of the timeline order 
allows us to (i) keep the untimed structure of the formulae the same (unlike in the transference 
theorem of pQ); (ii) show transference of a rich class of global timing constraints using freeze 
quantifiers (rather than only for the standard bounded time quantifiers of MTL/MITL). However, 
for implementations where the timeline order is not preserved, we have to settle for the less stronger 
guarantees provided by [1]. The work of |13] . in terms of robustness, deals mainly with spatial 
robustness of STL; the only temporal disturbances considered are constant time-shifts for the 
entire signal where the entire signal is moved to the past, or to the future by the same amount. 
The Skorokhod metric incorporates time-shifts which are variable along the timeline. 

2 Preliminaries 

Traces. A (finite) trace or a signal vr : [Ti,Te] i—>■ 0 is a mapping from a finite closed interval 
[Tj,Te] of M+, with 0 < Tj < Tg, to some topological space 0. If 0 is a metric space, we refer to the 
associated metric as T)q. The time-domain of tt, denoted tdom(7r) is the time domain [Tj,Te] over 
which it is defined. The time-duration of tt, denoted as tlen(7r), is sup (tdom(7r)). The t-suffix of vr 

^ Instead of having two separate parameters r and e for time and state variation, we pre-scale time and the n state 
components with n -|- 1 constants, and have a single value quantifying closeness of the scaled traces. 

^ Informally, two signals x, y are (T, J, (r, e))-close if for each point x{t), there is a point y{t') with \t — t'\ < t such 
that T){x{t),y{t')) < e; and similarly for y{t). 



for t G tdom(7r), denoted by tt*, is the trace vr restricted to the interval (tdom(7r) n [t,tlen(7r)]. We 
denote by the prefix trace obtained from vr by restricting the domain to [Tj,Tg] C tdom(7r). 

Systems. A (continuous-time) system 21 : i-)- Oip^ i-)- i-> Oop^, where is the set 

of finite closed intervals of M+, transforms input traces vTip : [Ti,Te] i—>• Oip into output traces 
TTop : [Ti,Te] I—)■ Oop (over the same time domain). We require that if 2l(7rip) i—)• vTop, then for every 
mintdom(7r) < Tg < maxtdom(7r), the system 21 maps TTip^^, to TTop^j.,. Thus, we only consider 
causal systems. Common examples of such systems are (causal) dynamical, and hybrid dynamical 
systems 

Conformance. A system 21^ conforms to the system 21 over an input trace vTip if 2l'(7rip) = 2t(7rip), 
i.e. if the behavior of 21' on the input trace vrip is the same as that of 21. The system 21' conforms to 
the system 21 over the input trace set TTip if conformance holds for each input trace in 77|p. Given 
a metric D over input traces, and an input trace set TTip, the quantitative conformance between 21' 
and 21 over flip is defined as the quantity sup^.^g^j.^ T (21' (vTip), 21 (vTip)). If ilip is the set of all input 
traces, this quantity is the distance between the two systems. 

Retimings. A retiming r : / i—?• /', for closed intervals 1,1' of M+ is an order-preserving (i.e. 
monotone) continuous bijective function from I to thus \it <t' then r{t) < r(t'). Let the class of 
retiming functions from I to I' be denoted as and let X be the identity retiming. Intuitively, 

retiming can be thought of as follows: imagine a stretchable and compressible timeline; a retiming 
of the original timeline gives a new timeline where some parts have been stretched, and some 
compressed, without the timeline having been broken. Given a trace vr : 0, and a retiming 

r : / I—)• Jjr; the function vr o r is another trace from / to 0. 

Definition 1 (Skorokhod Metric). Given a retiming r : I I' , let \ \ r —X||sup he defined as 
II r —X||sup = supig/ I r(t) — t|. Given two traces tt : I-^ 0 and vr' : Itt' t 0, where 0 is a metric 

space with the associated metrie Tq, and a retiming r : Ijr i—>• 1,^', let ||7r — vr' o r||ggjp he defined as: 

Ik “ ^'°''||^^p = suptg^^To(7r(t) , 7r'(r(t)) ). 

The Skorokhod distanc^ between the traces 7r() and vr'O is defined to be: 

Ts(7r,7r')= inf max(||r-X|| , ||7r - vr'o r|| ). □ (1) 

Intuitively, the Skorokhod distance incorporates two components: the first component quantifies 
the timing discrepancy of the timing distortion required to “match” two traces, and the second 
quantifies the value mismatch (in the metric space 0) of the values under the timing distortion. In 
the retimed trace vr o r, we see exactly the same values as in vr, in exactly the same order, but the 
times at which the value are seen can be different. 

Polygonal Traces. A polygonal trace tt : 0 where 0 is a vector space with the scalar field M 

is a continuous trace such that there exists a finite sequence min 1-,^ = to < ti <■■■< tm = max 
of time-points such that the trace segment between t^ and t^+i is affine for all 0 < A: < m, i.e., 
for tk < t < tk+i we have 7r(t) = 7r(tfc) -|- ■ (T^itk+i) — 7r(tfc)). Polygonal traces are obtained 

when discrete-time traces are completed by linear interpolation. We remark that after retiming, 
the retimed trace vr o r need not be piecewise linear (see e.g. |22j). 

Theorem 1 (Computing the Distance between Polygonal Traces [23] L Let tt : i— 

and vr' : 1^^/ i—)■ M” he two polygonal traces with mn and mjr' affine segments respectively. Let the 
Skorokhod distance between them (for the L 2 norm on W) be denoted as X)s(7r, vr'). 

^ The two components of the Skorokhod distance (the retiming, and the value difference components) can be weighed 
with different weights - this simply corresponds to a change of scale. 






1. Given 5 > 0, it can he checked whether tt') < 6 in time O 

2. Suppose we restrict retimings to he such that the i-th affine segment of tt can only be matched to 

affine segments i—W through i+W for all i, where W >1. Under this retiming restriction, we 
can determine, with a streaming algorithm, whether T)g(7r,TT') < 5 in timeO + m-,ri)-n-W). 

n 

Let us denote by (vr, tt') the Skorokhod difference between tt, tt' under the retiming restriction 
of the second part of Theorem[^ i.e., the value obtained by restricting the retimings in Equation [i|f| 
The value (vr, vr') is an upper bound on T)g{'K, tt'). In addition, for W' < W, we have (tt, tt') < 

3 Skorokhod Distance based Conformance Testing 

In conformance testing, we test for the variance in behavior of two given systems 2ti and 212 under 
the same inpulQ Given the same input, the two systems produce potentially differing output traces; 
the goal is to quantify this difference, and to determine an input signal that causes the corresponding 
output signals to exceed a user provided bound on the maximum tolerable output trace distance. 


Algorithm 1: Algorithm to test if maxTs(yi,y 2 ) < d 

yi,y2 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 


Input: Systems 2li, SI 2 , Bound S, Input Parameterization {P,F, B), Time Horizon T 
Output: u{t) s.t. yi = j /2 = 2t2(M), and Tgfyi, yz) > 5 

u random(P, F, B) 
maxCost < - 00 , m 0 

while (maxCost < 5) or (m < maxiterations) do 
2/1 simulate(Mi, u, T) 

2/2 simulate(M 2 , u, T) 

cost 'Ds{yi,y 2 ) 

if cost > maxCost then maxCost cost 
u •<— pickNewInputs(cost) 
m ■<— m + 1 

end 


Algorithm is a standard optimization-guided testing algorithm in which we have used the 
Skorokhod distance between two output traces as the cost function. In such algorithms, it is common 
to define a finite parameterization of the input space, represented by the tuple {P,F,B), where 
P = {pi,... ,pk} represents a set of parameters, F = {fi, ■ ■ ■, fk} represents a finite set of basis 
functions from [0,r] to M”, where T is some finite time-horizon, and for each pi G P, there is a 
hi ^ B that is a closed interval in M over which pi is assumed to take values. An input signal u is 
defined such that, for all t, u{t) = YliPi ' ^ valid input signal has the property that for all i, 

Pi G hi. 

In each step, the algorithm picks an input signal u and computes the Skorokhod distance 
between the corresponding outputs yi = 2li(n) and 2/2 = 212(tt). Based on heuristics that rely on 
the current cost, and a possibly bounded history of costs, the procedure then picks a new value for 
tt. For instance, in a gradient-ascent based procedure, the new value of tt is chosen by estimating the 
local gradient in each direction in the input-parameter space, and then picking the direction that 

^ is not a metric over traces (the triangle inequality fails). 

® It is also possible to extend our approach to allow inputs that are within some bounded Skorokhod distance. 







has the largest (positive) gradient. In our implementation, we use the Nelder-Mead (or nonlinear 
simplex) algorithm. 

The algorithm terminates when a violation is found (i.e., a pair of inputs that exceed the user- 
provided Skorokhod distance bound), or when the number of iterations is exhausted. The Skorokhod 
distance bound 6 is chosen based on engineering requirements, e.g., based on the maximum allowed 
weakening of the temporal logical properties that have been verified/tested on one system. 
Sampling and Polygonal Approximations. In practice, the output behaviors of the systems 
are observed with a sampling process, thus in implementations of Algorithmentities yi and y 2 on 
lines 4, 5 are time-sampled output trace sequences, from which the Skorokhod distance algorithm 
of Theorem constructs (continuous time) signals using linear interpolation. Given a timed trace 
sequence tseq, let |tseq]Li denote the continuous time trace obtained from tseq by linear interpo¬ 
lation. Let tseq,,., tseq^/ be two corresponding samplings of the traces vr, tt'. Since the Skorokhod 
distance is a metric, we have that 

I's(vr,7r') < Ds ([tseq^lLi, [tseq^,]Li) -b Ds ([tseq Jli, tt)- b T>s (Itseq^,]Li, vr') . 

If Asamerr is a bound Oil the distance between a trace, and an interpolated completion of its sam¬ 
pling, we have that T>g{7r, tt') < T)§ ([tseq^^-jLi, |tseq^/]Li) + 2-Asamerr- Thus, in a sampling framework, 
a value of 2-Asamerr needs to be added to the Skorokhod distance between the polygonal approxi¬ 
mations. 

Section [^presents a theory of (quantifiable) transference of logical properties. Section [^presents 
results on our implementation of Algorithm We also discuss several case studies, providing 
rationale for choosing the appropriate 6 value, and present results on the computation time and 
the conformance distance found. 

4 Transference of Logical Properties 

In this section, we demonstrate transference of logical properties. If two traces are at a distance 
of 5, and one trace satisfies a logical specification cf), we derive the “relaxation” needed (if any) 
in (j) so that the other trace also satisfies this relaxed logical specification. The logic we use is a 
version of the timed linear time logic TLTL j3] (a timed version of the logic LTL). We show that 
the Skorokhod distance provides robust transference of specifications in this logic: if the Skorokhod 
distance between two traces is small, they satisfy close TLTL formulae. We first present the results 
in a propositional framework, and then extend to ]R"'-valued spaces. 

4.1 The Logic TLTL 

Let "P be a set of propositions. A propositional trace it over P is a trace where the topological space 
is 2^, with the associated metric: D'p{a,a') = oo a ^ a', and 0 otherwise for a,a' G 2^. We 
restrict our attention to propositional traces with finite variability: we require that there exists a 
finite partition of tdom(7r) into disjoint subintervals Jq, p,..., Im such that vr is constant on each 
subinterval. The set of all timed propositional traces over P is denoted by P(P). 

Definition 2 (TLTL(JV) Syntax). Given a set of propositions V, a set of (time) variables Vj, 
and a set Pj of functions from to M, the formulae of TLTLp-Vj defined by the following 
grammar. 

f ■.= p\ TRUE I fj{x) ~ 0 I -■(/) I 01 A (/>2 I 01 V (/>2 I 01P i?i >2 I x.(j) where 

— p G V and x G Vj, and x = (xi, ... ,xi) with Xi G Vj for all 1 < i < 1; 

— fj G Pt is a real-valued function, and ~ is one of {<, <, >, >}. 


n 


We say that the variable x is bound in cj) ii (f) is x.'F, otherwise it is free. The quantifier “x.” is 
known as the freeze quantifier, and binds the variable x to the current time. A formula is closed if 
it has no free variables. 

Definition 3 (TLTLfJV) Semantics). Let TT : 7 I—7- 2^ 6e a timed propositional trace, to = 
min(7), and let £ : V I be the time environment mapping the variables in V to time values 
in I. The satisfaction of the timed sequence vr with respect to the TLTLfTV) formula (f in the 
time environment £ is written as vr \=£ cf, and is defined inductively as follows (denoting to = 
min tdom(7r)). 

TT |=£ pforpGV iffp G 7r(to); tt \=£ true; tt \=£ iff -it ^£ T; 

TT \=e 4>i A 02 tt \=£ 01 and tt \=£ 02; tt \=£ 0i V 02 iffir \=£ 0i or it \=£ 02; 

TT \=£ hixi,.. . ~ 0 iff fT{£{xi),.. .,£{xi)) r^O for ~G {<, <,>, >}; 

7T\=£X.Tf iff TT\=£[a::=to]'(^ wherc £ [x'.=to] agrees with £ fov all z X, and maps x to to] 

\=E 01 Id 02 iff TT* \=£ 02 for some t £ I and vr* \=£ 0i V 02 for all to < t' < t. 

A timed trace vr is said to satisfy the closed formula 0 (written as tt \= cf) if there is some environ¬ 
ment £ such that vr \=£ 0. □ 

The definition of additional temporal operators in terms of these base operators is standard: 
the “eventually” operator 00 stands for true 7/ 0; and the “always” operator 00 stands for -iO“'0. 
TLTL(JG) provides a richer framework than MTL |2T] for expressing timing constraints as: (i) freeze 
quantifiers allow specification of constraints between distant contexts, which the bounded temporal 
operators in MTL cannot do; and (ii) the predicates /t() ~ 0 for fj £ Fj allow the specification 
of complex timing requirements not expressible in MTL. 

Example 1 (Freeze quantifiers; TLTL(Jt) subsumes MTL0 Let Fj be the set of two variable 
functions of the form f{x, y) = x — y + c where c is a rational constant. Then TLTLfJV) subsumes 
MTL. The MTL formula QIA [a,b]di can be written as 

X. y.((y < X + 6) A (y > X + a) A 7?)^. 

We explain the formula as follows. We assign the “current” time tx to the variable x, and some 
future time ty to the variable y. The values tx and ty are such that at time ty, we have R to be 
true, and moreover, at all times between tx and ty, we have Q V 7? to be true. Furthermore, ty must 
be such that ty £ [tx + a, tx + b], which is specified by the term (y < x + 6) A (y > x + a). □ 

Example 2 (Temporal Constraints). Suppose we want to express that whenever the event Q occurs, 
it must be followed by a response R, and then by S. In addition, we have the following timing 
requirement: if £qr,srs,£qs are the time delays between Q and R, between 72,5, and between Q 
and 5 respectively, then: we must have + Egg < d for a given positive constant d. This 

can be written using freeze quantifiers as the TLTL formula 0: 

X. {Q -£ 0(y. (72 A 0 [ 2 . (5 A ((y - xf + (z - yf + {z - xf < d))]))) . □ 

4.2 Transference of TLTL Properties for Propositional Traces 

We show in this section that if a timed propositional trace vr satisfies a TLTLfJA) formula 0, then 
any timed trace vr' that is at most 6 distance away from vr satisfies a slightly relaxed version of the 
formula 0, the degree of relaxation being governed by 5] and the variance of the functions in Fj 
over the time interval containing the time domains of vr and vr*. 



Recall that the distance between two sets of propositions a, a' is oo if ct 7 ^ cr', and 0 if d = a'. 
The distance between two propositional traces is defined to be the Skorokhod distance with the 
aforementioned metric on 2 ^. 

Next, we define relaxations of TLTLfJV)formulae. The relaxations are defined as a syntac¬ 
tic transformation on formulae which do not have negations, except on the propositions. Every 
TLTLfJrlformula can be expressed in this negation-normal form. To remove negations from the 
until operator, we use the waiting for operator, W, defined as: 

TT \=E (fiiW 4>2 iff either (1) vr* \=£ (pi for all t G /; or (2) vr* \=£ (p2 for some t G /; and 
\=£ (pi V (p 2 for all to < t' < t. 

It can be showed that every TLTL(Jt) formula can be rewritten using the kV operator such that 
negations appear only over the propositions (the procedure is given in the Appendix). 


Definition 4 (J-relaxation of TLTLfJV) formulae). Let cp be a TLTL(JT) formula in which 
negations appear only on the propositional symbols. The 6 relaxation of cp (for 6>0) over a closed 
interval J, denoted rxj{(p), is defined as: 

rXj(TRUE) = TRUE 


rx‘ 

rx' 


'(P) 

r(“'P) 

j{(pi A <p 2 ) = ry(j{(pi) A rx^(</> 2 ) 


rx 


\rxj{x.'ip) 
J 


= p 

= -^P 
J 


= X. rx 


r(V') 


rXj( false) = FALSE 

rx^j\(pi V (p2) = rx5(</>i) V rx^j{(p2) 


Yx^j{(piU (P 2 ) = rx^j{(pi)U rx^j{(p 2 )\rxPj{(piyV (P 2 ) = rx^j{(pi)W ry.j{(p 2 ) 


rx^j{fj{xi,...,xi)) ^0) = 


where 




r hixi, ...,xi) -b ~ 0 

1 hixi, ~ 0 

[0, maxtdom(J) — mintdom(J)] 1 —)> M_|_, and 


-£{>,>} 


sup 
ti,... ,ti G J 
t'l,... ,t[ G J 


fT{t'i,---,t'i) 


s.t. \ti — t'f\ < 5 for all i 


( 2 ) 


Thus, instead of comparing the /t() values to 0, we relax by comparing instead to ±Kj_^{6). 
The other cases recursively relax the subformulae. The functions define the maximal change 

in the value of fj that can occur when the input variables can vary by 5. The role of J is the above 
definition is to restrict the domain of the freeze quantifier variables to the time interval J (from 
M+) in order to obtain the least possible relaxation on a given trace vr {e.g. we do not care about 
the values of a function in Tj outside of the domain tdom(7r) of the trace). 


Example 3 (6-relaxation for Bounded Temporal Operators - MTL ). We demonstrate how 5-relaxation 
operates on bounded time constraints through an example. Consider an MTL formula (p = QIA 
This can be written as a TLTL formula, and relaxed using the rx^^ function. The relaxed TLTL 
formula is again equivalent to an MTL formula, namely QU [a- 2 -S,b+ 2 -S]R- The details are explained 
in Example in the Appendix. □ 


Theorem 2 (Transference for Propositional Traces). Let he two timed propositional 
traces such that !D( 7 r, 7 r') < 6 for some finite 6. Let (p be a closed TLTL(JG) formula in negation- 
normal form. Ifir t (p, themr' 1= rx") (cp) where is the convex hull o/tdom(7r)Utdom(7r'). □ 

Theorem [^relaxes the freeze variables over the entire signal time-range /jr, 7 r'; it can be strength¬ 
ened by relaxing over a smaller range: if vr |= (p, and ti,... ,tk are time-stamp assignments to the 





freeze variables which witness vr satisfying (j), then Xi only needs to be relaxed over 

[ti — 6, ti + (5] rather than the larger interval /,r, 7 r'- These smaller relaxation intervals for the freeze 
variables can be incorporated in Equation We omit the details for ease of presentation. 


Example 4- Recall Exampleand the formula (j) presented in it. Suppose a flow vr satisfies (j); and 
let tt' be 6 close to vr under the Skorokhod metric (for propositional traces). Our robustness theorem 
ensures that (i) vr' will satisfy the same untimed formula Q —)■ 0 (R A OS)] and (ii) it gives a bound 
on how much the timing constraints need to be relaxed in cj) in order to ensure satisfaction by vr'; 
it states that vr' satisfies the following relaxed formula 4>' for every e > 0: 


vr = X. 


• (q 0(y. (r A 0 z. {s A (^{y - + {z- yf + {z - x)^ < 


where = d + 12 ■ (6 e)^ -\- 4\/3 • (5 + e) • ^/d. The constant is derived in the appendix. □ 


4.3 Transference of TLTL properties for ]R"^-valued Signals 

A timed -valued trace vr is a function from a closed interval I of M+ to M"". Eor a = ..., «”■) G 

M"", we denote the fe-th dimensional value as a[k]. The vr projected function onto the A:-th M 
dimension is denoted by vr^ : 1 1 —)• M. 

In order to define the satisfaction of TLTL formulae over timed M'^-valued sequences, we use 
booleanizing predicates p : M” i—>■ B, as in STL [13], to transform M"'-valued sequences in to timed 
propositional sequences. These predicates are part of the logical specification. In this work, we 
restrict our attention to traces and predicates such that each predicate varies only finitely often on 
the finite time traces under consideration. 

Definition 5 (TLTL(Jt,.As) Syntax). Given a set of variables Vj (the freeze variables), a set 
of ordered variables Es (the signal variables), and two sets iFj,iFs of functions, the formulae of 
TLTL(Jh,.Es) are defined by the grammar: 

4> := TRUE I /t(T) ~ 0 I fs{y) 0 I ->(j) I A (/)2 I V 02 I ^1 ^ ^2 I x.(j) where 

— X £ Vj, and x = (xi,..., x;) with Xj G Vj for all 1 < i < 1; 

- y = (yi, • • •, yd) with yj G Es for all 1 < j < d; 

— Ej and Es are disjoint; 

- fj G Ej and /$ G Es are real-valued functions, and ^ is or ^. D 

The semantics of TLTL(J't,.Es) is straightforward and similar to the propositional case (Def¬ 
inition]^. The only new ingredients are the booleanizing predicates fs{y) ~ 0: we define vr \=s 
/s(yi,---,yd) ~ 0 iff fsi'^ji[to],---,TTj^[to]) 0 for any freeze variable environment S, where 
to = mintdom(vr), and yt is the jj-th variable in Vs {i.e., yt refers to the ji-th dimension in the 
signal trace). We require that for a timed M"'-valued trace vr to satisfy 0, the arity of the functions 
in Es occurring in 0 should not be more than n, that is, functions should not refer to dimensions 
greater than n for an ME trace. 

6 relaxation of TLTL(Jh , Js)- Let Jvj be a mapping from Es to closed intervals of M, thus Jvsiz) 
denotes a sub-domain of z G Es. The relaxation function rxjj^ which operates on TLTL(Jt,Ts) 

formulae is defined analogous to the relaxation function rxj in Definition We omit the similar 
cases, and only present the new case for the predicates formed from Es (the full definition can be 
found in the appendix). 

rxj.Jvs (/s(^i,---,^z)) ~0) = 


fs{zi,...,zi) + Kf^{6) ~0 if ~G {>,>}; 
fs{zi,... ,zi) - Kf^{6) ~0 if ~G{<,<} 




where : [O, max^gVs I max Jvs(z) — min Jy 5 ( 2 ;)|] i—)> M+ is a function s.t. 

- 

fsiz'i, ■ ■ ■, z'l) 

for all i 


s.t. \zi — z'^ < 5 for all i > . 


The functions Kf^{S) define the maximal change in the value of fs that can occur when the input 
variables can vary by 5 over the intervals in 3vs{^) J. The role of Jy^ in the above definition is 
to restrict the domain of the signal variables in order to obtain the least possible relaxation bounds 
on the signal constraints; as was done in Definition for the freeze variables. 


Theorem 3 (Transference for M^-valued Traces). Let 7r,7r' he two W^-valued traces such the 
Skorokhod distance between them is less than 6 for some finite 6. Let be a closed TLTL(7h-,Ts) 
formula in negation-normal form. 7/vr 1= (/>, then tt' t rx^ j {(fi), where 

- 4,.' is the convex hull o/tdom(7r) Utdom(7r'); and 

— Iys(.2) is the convex hull of {7r{t)[k] \ t G tdom(7r)} U {4(t)[A:] | t G tdom(7r')}; where z is the 

k-th variable in the ordered set V^. □ 

Theorem can be strengthened similar to the strengthening mentioned for Theorem I] by 
relaxing the variables over smaller intervals obtained from assignments to variables which witness 

TT 4 (/>. 


Example 5 (Spatial Constraints and Transference). Recall Example snppose that the events 
Q, R, S are defined by the following predicates over real variables oi and 02 - Let Q = ai + 10-a2 > 3; 
the predicate 7? = |ai| + |a2| < 20; and 5 = |ai| + |a2| <15. Let vr satisfy this formnla with these 
predicates, and let vr' be 6 close to tt, for a finite 6 nnder the Skorokhod metric for M^. Onr robustness 
theorem ensures that tt' will satisfy the relaxed formula 


• {Q^ ^ 0(y. (r^ a 0 [z. A ((y - xf + {z- yf + {z-xf <d+ 12-5^)^ 


where the relaxed predicates Q^,R^,S^ are defined as follows; = oi + 10-02 > 3 
R^ = |ai| + I02I < 20 + 4 - 6 ; and = |oi| + |a2| < 15 + 4 - 6 . 



— 22-(5; and 


5 Experimental Evaluation 

Skorokhod Distance Computation Benchmark. The Skorokhod distance is computed with 
the help of a streaming, sliding window monitoring rontine which checks for a fixed 5 whether the 
linear interpolations of two time-sampled traces are at most 5 away from each other. The least snch 
5 value is computed by binary search over the monitoring routine. The upper limit of the search 
range is set to the pointwise metric {i.e assuming the identity retiming) between the two traces. The 
traces to the Skorokhod procedure are pre-scaled, each dimension (and the time-stamp) is scaled 
by a different constant. The constants are chosen so that after scaling, one nnit of deviation in one 
dimension is as nndesirable as one unit of jitter in other dimensions. We next present a benchmark 
on the distance computing routine. 

Consider the hybrid dynamical system 2li shown in Fig. The system consists of two water 
tanks, each with an ontlet from which water drains at a constant rate dj. Both tanks share a single 
inlet pipe that is switched between the tanks, filling only one tank at any given time at a constant 
inflow rate of i. When the water-level in tank j falls below level ij, the pipe switches to fill it. The 






/ l 2 < (-2 



h\ < 


Fig. 1. System 2li used for benchmarking Skorokhod Distance computation. Inflow rate i, Drain rate di for tank 1 
and d 2 for tank 2 are all inputs to the system. 

Table 1. Benchmarking the computation of Ts(7ri,7r2), where tti is a trace of system 2li described in Fig. and 
7r2 is a trace of system 212, which is 2li with an actuation delay. ®2 is the naive pointwise distance. Both tti and 7r2 
contain equally spaced 2001 time points over a simulation horizon of 100 seconds. 


Window size 

Avg. Ts 

Avg. Time taken (secs) 

Do — 2 ^ 9 

max ^ 



Computation 

Monitoring 


20 

8.58 

0.81 

0.13 

0.09 

40 

8.35 

1.55 

0.26 

0.18 

60 

8.09 

2.31 

0.39 

0.26 

80 

7.88 

3.05 

0.52 

0.33 

100 

7.72 

3.77 

0.64 

0.38 


drain and inflow rates di, d 2 and i are assumed to be inputs to the system. Now consider a version 
2 I 2 that incorporates an actuation delay that is a function of the inflow rate. This means that 
after the level drops to ij for tank j, the inlet pipe starts filling it only after a finite time. 2ti and 
2t2 have the same initial water level. We perform a fixed number of simulations by systematically 
choosing drain and inflow rates di, d 2 , i to generate traces (water-level vs. time) of both systems 
and compute their Skorokhod distance. We summarize the results in Table 

Recall that !)§ (the Skorokhod distance) computation involves a sequence of monitoring calls 
with different 6 values picked by a bisection-search procedure. Thus, the total time to compute Rg 
is the sum over the computation times for individual monitoring calls plus some bookkeeping. In 
Table we make a distinction between the average time to monitor traces (given a 6 value), and 
the average time to compute T>$. There are an average of 6 monitoring calls per Rg computation. 
We ran 64 simulations by choosing different input values, and then computing Rg for increasing 
window sizes. As the window size increases, the average Rg is seen to decrease; this is expected as 
a better match may be achieved in a larger window. The computation time is also seen to increase 
linearly, as postulated by Theorem [Tj Finally, we see that the Skorokhod distance is less aggressive 
at classifying traces as distant (as shown by its lower overall numbers) than a simpler metric R 2 
(defined as as the maximum of the pointwise L 2 norrcj^ . We can see this discrepancy becomes more 
prominent with increased window size (because of better matches being available). 

Case Study: LQR-based Controller. The first case study is an example of an aircraft pitch 
control application taken from the openly accessible control tutorials for Matlab and Simulink [25]. 
The authors describe a linear dynamical system of the form: x = (A — BK)x + BOdes- Here, x 
describes the vector of continuous state variables and Odes is the desired reference provided as an 
external input. One of the states in the x vector is the pitch angle 9, which is also the system 
output. The controller gain matrix K is computed using the linear quadratic regulator method |5|, 

® Even though the difference is only 38% with respect to the pointwise metric, this difference is amplified in the 

original state value domain, as in the experiment, the inpnt state values to the Skorokhod routine were scaled by 

0 . 1 . 





















Table 2. Variation in Skorokhod Distance with changing sampling time for an aircraft pitch control system with an 
LQR-based controller. Time taken indicates the total time spent in computing the upper bound on the Skorokhod 
distance across all simulations. We scale the signals such that a time-jitter of 0.5 seconds, is treated the same as a 
value-difference of 0.08 radians, and the window size chosen is 150. The system is simulated for 5 seconds, with a 
variable-step solver. 


Controller 

Sample-Time 
(seconds) 

Skorokhod 

distance 

Time taken (seconds) 
to compute T>s 

Number of 
simulations 

0.01 

0.012 

232 

104 

0.05 

0.049 

96 

104 

0.1 

0.11 

70 

106 

0.3 

0.39 

45 

104 

0.5 

1.51 

40 

101 


a standard technique from optimal control. We are interested in studying a digital implementation 
of the continuous-time controller obtained using the LQR method. To do so, we consider sampled- 
data control where the controller samples the plant output, computes, and provides the control 
input to the plant every A seconds. To model sensor delay, we add a fixed delay element to the 
system; thus, the overall system now represents a delay-differential equation. 

Control engineers are typically interested in the step response of a system. In particular, quan¬ 
tities such as the overshoot/undershoot of the output signal (maximum positive/negative deviation 
from a reference value) and the settling time (time it takes for transient behaviors to converge to 
some small region around the reference value) are of interest. Given a settling time and overshoot 
for the first system, we would like the second system to display similar characteristics. We remark 
that both of these properties can be expressed in STL, see [T9j for details. We quantify system 
conformance (and thereby adherence to requirements) in terms of the Skorokhod distance, or, in 
other words, maximum permitted time/space-jitter value <5. For this system, we know that at nom¬ 
inal conditions, the settling time is approximately 2.5 seconds, and that we can tolerate an increase 
in settling time of about 0.5 seconds. Thus, we chose a time-saling factor of 2 = We observe 
that the range of 0 is about 0.4 radians, and specify an overshoot of 20% of this range as being 
permissible. Thus, we pick a scaling factor of 0.08 for the signal domain. In other words, Skorokhod 
distance 5 = 1 corresponds to either a time-jitter of 0.5 seconds, or a space-discrepancy of 0.08 
radians. 

We summarize the results of conformance testing for different values of sampling time A in 
Table It is clear that the conformance of the systems decreases with increasing A (which is to 
be expected). The time taken to compute the Skorokhod distance decreases with increasing A, as 
the number of time-points in the two traces decreases. 

Case Study: Air-Fuel Ratio Controller. In |T9] , the authors present three systems representing 
an air-fuel ratio (A) controller for a gasoline engine, that regulate A to a given reference value of 
Aref = 14.7. Of interest to us are the second and the third systems. The former has a continuous¬ 
time plant model with highly nonlinear dynamics, and a discrete-time controller model. In |20j . 
the authors present a version of this system where the controller is also continuous. We take this 
to be 2li. The third system in |19j is a continuous-time closed-loop system where all the system 
differential equations have right-hand-sides that are polynomial approximations of the nonlinear 
dynamics in 2li. We call this polynomial dynamical system 212- The rationale for these system 
versions is as follows: existing formal methods tools cannot reason about highly nonlinear dynamical 
systems, but tools such as Flow* [9], C2E2 and CORA |3] demonstrate good capabilities for 





Table 3. Conformance testing for closed-loop A/F ratio controller at different engine speeds. We scale the signals 
such that 0.5 seconds of time-jitter is treated equivalent to 10% of the steady-state value (14.7) of the A/F ratio 
signal. The simulation traces correspond to a time horizon of 10 seconds, and the window size is 300. 


Engine 
speed (rpm) 

Skorokhod 

distance 

Computation 
Time (secs) 

Total Time 
Taken (secs) 

Number of 
simulations 

1000 

0.31 

218 

544 

700 

1500 

0.20 

240 

553 

700 

2000 

0.27 

223 

532 

700 


polynomial dynamical systems. Thus, the hope is to analyze the simpler systems instead. In |19j . 
the authors comment that the system transformations are not accompanied by formal guarantees. 
By quantifying the difference in the system behaviors, we hope to show that if the system 2 I 2 
satishes the temporal requirements (p presented in m, then 2li satishes a moderate relaxation of 
(f. We pick a scaling factor of 2 for the time domain, as a time-jitter of 0.5 seconds is the maximum 
deviation we wish to tolerate in the settling time, and pick 0.68 = ^ ^ as the scaling factor for 

A (which corresponds to the worst case tolerated discrepancy in the overshoot). 


The results of conformance testing for these systems are summarized in Table In |12) , the 
authors posed a challenge problem for conformance testing. In it, the authors reported that the orig¬ 
inal nonlinear system and the approximate polynomial system both satisfy the STL requirements 
specifying overshoot/undershoot and settling time. We, however, found an input that causes the 
outputs of the two systems to have a high Skorokhod distance. Thus, comparing the two systems 
by considering equi-satisfaction of a given set of STL requirements such as overshoot/undershoot 
and settling time may not always be sufficient, and our experiment indicates that the more nuanced 
Skorokhod metric may be a better measure of conformance. 


Case Study: Engine Timing Model. The Simulink demo palette presented by the Mathworks 
j24| contains a system representing a four-cylinder spark ignition internal combustion engine based 
on a model by Crossley and Cook |in) . This system is then enhanced by adding a proportional plus 
integral (P-|-I) control law. The integrator is used to adjust the steady-state throttle as the desired 
engine speed set-point changes, and the proportional term compensates for phase lag introduced by 
the integrator. In an actual implementation of such a system, such a P-l-I controller is implemented 
using a discrete-time integrator. Such integrator blocks are typically associated with a particular 
numerical integration technique, e.g., forward-Euler, backward-Euler, trapezoidal, etc. It is expected 
for different numerical techniques to produce slight variation in the results, and we wish to quantify 
the effect of using different numerical integrators in a closed-loop setting. We try to check if the 
user-provided bound of 5 = 1.0 is satisfied by systems 2ti and 2 I 2 , where 2ti is the original system 
provided at [23] , while 2 I 2 is a modified system that uses the backward Euler method to compute the 
discrete-time integral in the controller. We try to determine the input signal that leads to a violation 
of this 5 bound, using a simulation-guided approach as described before. We scale the outputs in 
such a way that a value discrepancy of 1% of the the output range ( 1000) is equivalent to a time 
discrepancy of 0.1 seconds. These values are chosen to bias the search towards finding signals that 
have a small time jitter. This is an interesting scenario for this case study where the two systems 
are exactly equivalent except for the underlying numerical integration solver. We find the signal 
shown in Fig. for which we find output traces with Skorokhod distance 1.04. The experiment 
uses 296 simulations and the total time taken to find the counterexample is 677 seconds. 







Fig. 2. Example of non-conformant behavior found using a simulation-guided optimization algorithm with the Sko- 
rokhod distance between system output trajectories as the cost function. 

6 Conclusion 

Metrics for comparing behaviors of dynamical systems which quantify both time and value dis¬ 
tortions have heretofore been an object of mathematical inquiry, without enough attention being 
paid to computational aspects and connections to logical requirements. We argue that the Sko- 
rokhod metric provides a robust definition of conformance by proving transference of a rich class of 
temporal logic properties. We also demonstrate the computationally tractability of the metric for 
practical use by constructing a conformance testing tool in a simulation and optimization guided 
approach for finding and quantifying non-conformant behavior of dynamical systems. Pinpointing 
the source of trace deviations is necessary in many engineering applications; our tool allows for 
independent weighing of time and value-dimension distortions in order to achieve this objective. 
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Appendix 


A. Transference Formalism and Proofs 


Example 6 (Freeze Quantification). Suppose we want to express that whenever the event Q occurs, 
it is followed later by R, and then by S, such that the time difference between occurrences of Q 
and R is at most 5, and also the time difference between occurrences of Q and S is at most 10. 
This can be expressed in TLTL(JV) as 

O^x.Q (}(^y.[R A {y < X + 5) A (} (z. (S A z < X + 10)) j)^ 

Thus, freeze quantification, by giving a mechanism to bind times to variables, allows us to relate, 
with several constraints, far apart events. □ 


Example 7 (Freeze Quantification Functions). Suppose we want to express that whenever the event 
Q occurs, it must be followed by a response R within time for some A > 1 where tg is the time 
at which Q occurred; thus, the later Q occurs the more delay we can tolerate in the response time. 
The requirement can be expressed as x. [Q ^ ()[y. {R A 0 < y < A®))). □ 


Example 8 (6-relaxation for Bounded Temporal Operators - MTL ). We demonstrate how 5-relaxation 
operates on bounded time constraints through an example. Consider an MTL formula = QU [a,6] 7?. 
The 5-relaxation of this formula over the closed interval Ij^j = M+ \s QIA [a- 2 - 5 ,h+ 2 - 5 ]R- This can 
be seen as follows. The formula 4> can be written in TLTL syntax as: 


x.QUy. {{y < X -^h) A{y > x -^h) A R). 
The 5-relaxation of this formula according to Definition i 


is: 


rxm 


{x.QU y. {{y < x b) A {y > x a) A R)) = 


— ''4+ {x.QU y. {{y — x — b<0)A{y — x — a>0)A R)) 


= x.QU y. 


(y — X — 6 — 2-5 < 0) A 

{y — X — a-\-2 -5>t))AR^ 

since the Lipschitz constant of y — x — c is 2 
for any constant c 

= x.QU y. ((y < x -|- 5 -|- 2-5) A (y > x -|- a — 2-5) A i?) 

= QU [a-2.&,b+2.5]R- 

Thus, the time constraint interval boundaries are relaxed by 2-5. The factor of 2 arises because 
there are two contributing factors: the starting time of Q can be “pulled back” by 5, and the time 
of R can be postponed by 5; thus, the time duration in between Q and R increases by 2-5. □ 


Removing Negation nsing the W Operator. The following identities hold relating the W 
operator to the U operator 

1 . 4>iU (t >2 = -■ (-'(</>2) W (-k/)! A-'(/)2)); and 

2. <^i>V<^2 = ^i^ih)Ui^<PiA^fi2)). 

Informally, the first identity states that -<{(l)iU 4>2) holds iff either (i) (f >2 never holds; or (ii) there 
is a point where is false, and at that point and all points before it, (f >2 has remained false. The 


second identity is similar. The first identity above allows us to “push” the negations down using 
the W operator. The mechanism for the three interesting cases is below. 

^ (/t(xi, ... ,a:z) ~ 0) = /t(xi, ...,x;) neg(~)0, 

where, for ~ G {<, <, >, >} we have 
neg(<) to be >; neg(<) to be >; 

neg(>) to be <; neg(>) to be < 

-i(x.V') = x.-'(V') 

{(j)lU (j)2) = w A-.(/)2) 

Proposition 1. The function rx is a relaxation on TLTL(JV) formulae, i.e. if a timed propositional 
trace \= (f for a TLTL(Jt) formula cf, then tt |= {(p). 

Proof. Observe that, over the predicates /t(xi, ... ,Xi) ~ 0, the function rx is indeed a relaxation, 
i.e, if fT{ti,... ,ti) ~ 0 for values ti,... ,ti, then rx^^ {friti, ■ ■ ■ ,ti)) ~ 0) also holds. The result 
follows by a straightforward induction argument. □ 

Proof of Theorem Let untime(i^) be the formula where all freeze variable constraints are re¬ 
placed by TRUE {e.g. untime(x.(Q A x < 5)) is x.{Q A true)). Since T(7r,7r') < 6, we have that 

there exists a retiming r : tdom(7r) i—)• tdom(7r') such that 

7r(t) = 7r'(r(t)). (3) 

This implies that both vr and tt' satisfy untime(())), which can be shown by an induction argument. 
The interesting cases are for the U and W operators. We sketch the argument for the Id case 
(the argument for W is similar). The time environment £' for vr' assigns the time r(tx) to the 
freeze variable x where the witnessing freeze variable environment £ for tt \= 4> assigns tx to x. Let 
\=s 4>ild (j) 2 : and let t be the time value which demonstrates this satisfaction (as in Definition [^ , 
with the corresponding freeze variable environment £. To show vr' \=£i (f>iU (p 2 , we pick the time 
r(f), with the environment £' for vr' which assigns the time r{tx) to the freeze variable x where 
£{x) = tx. It can be checked that, due to Equation!^ we have (i) r{t) > £'{x), for a freeze variable 
X in (pi U (p 2 (which was previously bound); (i) vr'db (^ 2 ; and (ii) for all t'o < r(i)> we have 
\=£' 4>i V (p 2 - Thus, r(t), and £' demonstrate that vr' \=£i (p\U cp 2 . 

We now check what is the relaxation needed on the original freeze variable constraints so that 
vr' satisfies the relaxed constraints. Without loss of generality, assume that each freeze variable x is 
only quantified once in (p, i.e. once it is bound to a value by “x.”, it is not “re-bound” with another 
application of “x.”. 

Let vcjr denote an assignment of time values (from I) to the freeze variables such that all the freeze 
variable constraints in (p are satished, i.e. is an time environment witness to the satisfaction of (p 
by vr. Consider a free variable assignment corresponding to where Kt^i{x) = r(K 7 r(x)). This 
is a legal variable assignment compatible with some lA , W time witnesses which demonstrate that 
vr' satisfies untime(i?!)), as shown previously. Observe that by the existence of a retiming function, 
for all freeze variables x occurring in cp, we have that \kt^i{x) — K 7 r(x)| < 5. 

Since the time values of variables are different in and Ktt', the original freeze constraints {e.g. 
X < 5) in (p might not be satisfied with the assignment Consider a freeze variable constraint 
/t(xi, ..., xz) ~ 0 in 0. We know that /T(«^7r(xi),... ,Kn{xi)) ~ 0 is true. As |Kjr'(x) — K 7 r(x)| < 6 
for all freeze variables x occurring in cp, by the definition of relaxation, we have that 


1- hinnixi), k-k{xi)) + Kj{5) ~ 0 if ~ G {>, >}; and 
2. fj{KT,{xi), ..., Kn{xi)) - Kj{6) ~ 0 if ~ G {<, <}. 

This implies that Kt^i is also a witness to the satisfaction of rx^ {(j)) by tt'. Thus, vr' |= rx^ (</)). □ 

^ TT . TT^ TT . -TT^ 


Example^ details. Since tt satisfies cj), we must have time-stamps tx,ty,tz bound to x,y,z re¬ 
spectively so that with these assignments, the formula (j) is satisfied. Since vr' is 5 close to tt, for 
every e > 0, there is a retiming from tt to tt' such that the times tx,ty, tz in tt are mapped to 
in vr' such that (a) \tx — t'x\ < 5-|- e; and (b) \ty — t'y\ < (5 -|- e; and (c) < 5 + e. Let 5' = 5 + e. 

The sum {t'x - t'yf + {t'y - - t'xf is 

~ ((^X ~ ^x) + {lx ~ ly) + {ty ~ ty)) + {(ty ~ ty) + {ty — tz) + {tz — t^)) + 

((^2 ~ ^ 2 ) + {tz ~ tx) + {tx — tx)) 

— 2 {{t'x — tx)^ + {t'y — ty)‘^ -|- {t'^ — tz)"^) + {tx — ty)‘^ + {ty — -|- {tz ” T 

2 {{tx ~ tx){tx — ty) + {ty — ty){tx — ty) + {t^ — tx){ty — ty)^ + 

^ {{^y ~ ^y'^{^y ~ ~ tz){ty — tz) -I- {ty — ty){tz — t^,)) -I- 

2 {{tz ~ tz){tz ~ tx) + {tx ~ tx){tz ~ tx) + {tz ~ tz){tx ~ tx)) 

< 66'^ + d + 26'\tx - ty\ + 2(5'2 + 26'\ty -tz\ + 25'^ + 26'\tz - tx\ + 25'^ 

= d 126'^ 4:5' {\tx — ty\ -\- \ty — tz\ -|- |t2 ~ tx | ) 

< d+l2-5'^ + 4 V 3 -5' -Vd 


In the last step above, we use the inequality: |a| -|- |6| -|- |c| < \/3 • + &^ + This inequality is 

obtained by applying the Cauchy-Schwarz inequality to the tuples (|a|, \b\, |c|) and (1,1,1). Thus, 
by Theorem for every e > 0, we have 


vr' \= x.(^Q ^ 0{y- (^RAO z.(^S a (^{y - x)^ + {z - + {z - x)^ < 


where = d -|- 12 • <5'^ -|- 4\/3 ■ 5' ■ y/d. 


□ 


Definition 6 (d-relaxation of TLTL(Jt,Ts) formulae). Let (j) be a TLTL(Jt,Ts) formula in 
which negations appear only on the prepositional symbols . The 5 relaxation of cf (for 6 > 0), 
denoted rxj^ {f) is defined as follows, where a closed subset ofreals^, is the domain of the 
variables in Vj; and lyj is a mapping from Vs to closed intervals o/M such that Iys(.?) denotes the 





domain of z. 


(true) = true; 


rxf I (false) = false; 


'■’‘U.I.i' 


h A (j) 2 ) = A rxsif) 2 )-, 

h V (1)2) = rxf^^,i^^((/>i) V rx^^^,i^^((/> 2 ); 
^ 1.3 W; 

.^^,1^3 (<^ 1 ^<^ 2 ) = r4^^jy^i4>2y, 




i-x/^ 

rx/^^,lvs ^ ^ ''xf^T.1.3 (‘('1) ^ (*^2) 

f fuAi, ■■.,zi) + Kf^{6) ~ 0 
\fuizi, ■ ■ ■ ,zi) - Kf^{6) ~ 0 


rx/^^Jvs ifu{zi,...,zi)) ~ 0) = 


2 / -£{>,>}; 
2 / -£{<,<}; 


where U G {T,S} mi/i Kj^ being as in Definition\^ 

andKf^ : [O, max | max( 2 ;) — mmlv 5 (z)|] 1 —)• M+ 
z^Vs 

is a function such that: 
fs{zi,..., zi) 


Kf^{5) = sup 

2 :i e ivsAi); z[ G 1 ^ 3 ( 2 ;') 
for all i 


f5{zy---,zl) 


s.t. \zi — z[\ <5 for all i 


n 

The functions Kf^{6) define the maximal change in the value of fs that can occur when the 
input variables can vary by 5. The role of Iv "3 in the above definition is to restrict the domain of 
the signal variables in order to obtain the least possible bounds relaxation bounds on the signal 
constraints; as was done in Definition for the freeze variables. 

Proposition 2. The function rxj^ is a relaxation on TLTL(Jt,Ts) formulae, i.e. if a timed 
MT-valued trace tt \= (f for a TLTL(J't,Ts) formula cj), then tt ^ rx^^ (0). 

Proof. The proof is similar to the proof of Proposition □ 

Proof of Theorem The proof use the result for the propositional case, Theorem We con¬ 
struct the propositions pj^ defined to be rx|^ {fs{y)) ~ 0 ) for the constraints over Vs in the 
formula 0 ; and define the TLTL(Jt) formula (pp as that obtained from (p by syntactically replacing 
each constraint fs{y) ~ 0 in </> by p/ 5 . Let Vs denote all such predicates for cp. We obtain the timed 
Vs propositional traces Trp^,Vp^ from vr, vr' by mapping to propositions. By the definition of the 
skorokhod distance, the distance between vr-pj and is less than <5. By Theorem ^ (pp. 

This implies vr' |= rx^^ {(p). □ 

B. Details on Case Studies 

LQR-based pitch controller. The aircraft pitch controller system has 3 state variables, and the state 
vector X = [a q 6], where a is the angle of attack, q is the pitch rate, and 6 is the pitch angle. 




The system has a single input <5 (the elevator deflection angle). In deriving the control law, the 
designers use the state feedback law to substitute 5 = Odes — Kx, where Odes is the desired pitch 
angle. The resulting dynamical equations of the system are of the form x = (^ — BK)x + BOdes, 
and the output of the system is the state variable 0. Note that the K matrix is the gain matrix 
resulting from the LQR control design technique. The values of the A, B and K matrices are as 
given below: 



■-0.313 

56.7 

O' 


'0.232 

A = 

-0.0139 

-0.426 

0 

B = 

0.0203 


_0 

56.7 

0 


0 

K = 

-0.6435 

169.6950 

7.0711] 



Air-Fuel Ratio Controller. The Air-Fuel (A/F) ratio control systems that we consider are simplified 
versions of industrial-scale models. Both versions have 2 exogenous inputs, and 4 continuous states. 
The inputs are engine speed (measured in rpm) and the throttle angle (in degrees). The throttle 
angle is a user input, and it is common to assume a series of pulses or steps as throttle angle inputs. 
The engine speed is considered an input to avoid modeling parts of the powertrain dynamics. In 
our experiments, we typically hold the engine speed constant. This is to mimic a common engine 
testing scenario involving a dynamometer, which is a device to provide external torque to the engine 
to maintain it at a constant speed. Of the 4 continuous states, we assume that 2 of these states 
are from the plant model (that encapsulates physical processes within the engine), while 2 states 
belong to the controller. The plant states p and A denote intake manifold pressure and the A/F 
ratio respectively. The controller states pe denotes the estimated manifold pressure (with the use 
of an observer) used in the feed-forward control, and the state i denotes the integrator state in 
the P-|-I feedback control. We check conformance with respect to the system output A. For the 
dynamical system equations, please refer to gSEo] . 







